1. Field of the Invention
The present invention relates to combination processing of an image using multi-viewpoint image data.
2. Description of the Related Art
Conventionally, when an image was captured by a camera out of focus, it was necessary to recapture the image after performing focusing again. Further, when it was desired to obtain focused images of a plurality of subjects with different depths, it was necessary to capture the image of each subject in focus a plurality of times.
In recent years, a technique called light field photography has been developed, which is capable of adjusting the focus position in the later image processing (refocus) by adding a new optical element to the optical system to acquire images from multiple viewpoints.
By using this technique, there is an advantage that a failure in focusing at the time of image capturing can be made up by image processing because it is possible to perform focusing after image capturing. Further, there is also an advantage that it is possible to obtain a plurality of images focused on an arbitrary subject in an image from one captured image by changing the image processing method, and therefore, it is possible to reduce the number of times of image capturing.
In light field photography, the direction and intensity of a light beam that passes through each position (light field, hereinafter, referred to as “LF”) in a plurality of positions in a space are calculated from multi-viewpoint image data. Then, by using the information of the obtained LF, an image on the assumption that light passes through a virtual optical system and forms the image on a virtual sensor is calculated. By appropriately setting such a virtual optical system and a virtual sensor, refocus is also enabled. The mathematical nature and mathematical fundamentals relating to LF are discussed by R. NG et al. (R. NG, M. Levoy, M. Bredif, G. Duval, M. Horowitz, P. Hanrahan, “Light Field Photography with a Hand-held Plenoptic Camera” (Stanford Tech Report CTSR 2005-02, 2005). Hereinafter, processing to calculate image data that a virtual sensor obtains from multi-viewpoint image data is referred to as combination processing.
As an image capturing technique for acquiring LF, a camera array in which single cameras are arranged side by side and a Plenoptic camera in which a microlens array is arranged behind amain lens are known. Here, “single camera” represents one of the camera array which consists of a plurality of cameras. It is possible for the camera array and the Plenoptic camera to combine image data on the virtual sensor after image capturing from the acquired LF.
As a method for combining images on the virtual sensor from LF, there is known a method in which acquired image data of a plurality of images is projection-transformed onto a virtual sensor, and added and averaged (WO2008/050904).
Further, there is known a technique to put a telephoto image into a wide-angle image with the magnification adjusted to that at the time of zoom by capturing images using a camera array with different focal lengths in which a plurality of single cameras with different focal lengths exists mixedly to perform optical-equivalent zoom without mounting a zoom mechanism (Japanese Patent Laid-Open No. 2011-55246).
For example, a case is considered where the technique of WO2008/050904 is applied when performing refocus on multi-viewpoint image data captured by the camera array with different focal lengths shown in Japanese Patent Laid-Open No. 2011-55246. In this case, the image data captured by a single camerawith a different angle of view is not utilized, and therefore, the combined image that is completed is of relatively low image quality. Similarly, also in the case where a distance to a subject is estimated using multi-viewpoint image data captured by a camera array with different angle of view (or “focal lengths”), relatively low precision will result unless the image data captured by a single camera with a different angle of view is utilized.